Papers with BERT-based metrics
MENLI: Robust Evaluation Metrics from Natural Language Inference (2023.tacl-1)
Copied to clipboard
| Challenge: | Recent proposed BERT-based evaluation metrics for text generation are vulnerable to adversarial attacks, e.g., relating to information correctness. |
| Approach: | They propose to use BERT-based evaluation metrics for text generation to evaluate text for semantic similarity but are vulnerable to adversarial attacks using Natural Language Inference. |
| Outcome: | The proposed metrics outperform existing summarization metrics but perform below SOTA MT metrics on standard benchmarks. |
Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing models for dialog evaluation are trained using a single relevant response and multiple random negatives. |
| Approach: | They propose a dataset to test whether model-based dialog evaluation metrics can be used to train models . they propose n-gram based metrics and embedding based ones to be used for model-driven evaluation . |
| Outcome: | The proposed model outperforms existing models on a reddit dataset on relevant responses and adversarial responses. |
Rethinking Evaluation Metrics for Grammatical Error Correction: Why Use a Different Evaluation Process than Human? (2025.acl-short)
Copied to clipboard
| Challenge: | Existing automatic evaluation metrics are based on procedures that diverge from human evaluation. |
| Approach: | They propose to aggregate automatic evaluation metrics to bridge this gap . they propose to use edit-based metrics, -gram based metrics and sentence-level metrics to find the best ranking system. |
| Outcome: | The proposed method outperforms existing metrics on the SEEDA benchmark and improves edit-based metrics, -gram based metrics and sentence-level metrics. |
Reproducibility Issues for BERT-based Evaluation Metrics (2022.emnlp-main)
Copied to clipboard
| Challenge: | Reproducibility is of utmost concern in machine learning and natural language processing . lexical-overlap metrics are still the dominant metric in natural language generation . |
| Approach: | They ask whether results and claims from four recent BERT-based evaluation metrics can be reproduced. |
| Outcome: | The proposed metrics outperform the dominant metric, BLEU, and show that they can be reproduced. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |